For traditional SURF (Speeded-Up Robust Feature) algorithm takes a long time for constructing local feature descriptors, an improved SURF algorithm was proposed to meet the real-time requirement. Firstly, the Determinant of Hessian (DoH) matrix was adopted to detect the key points of an image. Non-maximum suppression algorithm and interpolation operation were used to search and position the extreme points. Secondly, gray centroid method was adopted to determine the main direction of the key points. Then a binary descriptor, BRIEF (Binary Robust Independent Elementary Feature), was adopted to describe the key points, and the main direction of the key points was used to construct a directed feature descriptor with the objective of keeping its rotation invariance. Finally, Hamming distance was used to preliminarily determine the matching points. Then, the mismatching points were removed to improve the matching accuracy by ratio detection method and RANSAC (Random Sample Consensus) algorithm. The experimental results show that, when the improved SURF algorithm is applied to the flexible clamping robot, the matching time is reduced from 214.10 ms to 86.29 ms, the matching accuracy is increased by 2.6% compared with traditional SURF algorithm. Therefore, the proposed algorithm can improve the workpiece image matching speed and matching precision of flexible clamping robot effectively.